Coding Theory

The field of coding theory emerged with the pioneering work of Claude E. Shannon “A Mathematical Theory of Communication”, published in 1948 in the Bell System Technical Journal. In his work, Shannon investigates both the transmission and compression of information. A probabilistic model is proposed for the communication channel or for the source, and a mathematical measure is proposed to quantify the amount of noise in the channel (capacity) or the amount of redundancy in the source (entropy). In each setting, the unit used to measure the amount of noise or redundancy is the bit (binary digit), a terminology attributed to John W. Tukey. The notion of code is also developed to protect information from the noise or to compress redundant information. Shannon then provides two monumental results: (1) the largest rate at which information can be reliably transmitted is given by the channel capacity, (2) the largest rate at which information can be reliably compressed is given by the source entropy. The notion of reliability requires formal definitions, discussed below, but in words refers to essentially not loosing information with high probability. These results are particularly powerful since they indicate at the same time what can be achieved with communication and compression protocols, and what cannot be achieved irrespectively of the algorithmic approach. This type of result is known as “information theoretic”. Shannon did however leave a major open problem with his theory. While he proves that there exists codes achieving rates up to the capacity, or up to the entropy, Shannon does not provide a construction of such codes. This is one of the first application of the “probabilistic method”, pioneered by Paul Erdös around the same time. This also gave birth to coding theory, whose major goal has been to filling in this gap, by constructing good codes with manageable complexities. By now, coding theory has also expended into a field of its own, with a broad spectrum of applications in electrical engineering, computer science and discrete mathematics. Shortly after Shannon’s seminal paper, Richard Hamming published in 1950 “Error Detecting and Error Correcting Codes”, also in the Bell System Technical Journal, which contains some of the fundamental concepts of coding theory. Hamming focused not on the probabilistic model of Shannon, but on a worst-case noise model. More specifically, Hamming developed coding schemes which can correct a fixed number of errors with probability one, as opposed to Shannon who contented himself to correct errors with high probability on the noise realizations. The goal of this note is to provide a few examples of fundamental coding theory results, in both the source and channel setting, and in both the worst-case and probabilistic setting.